THE VALIDITY OF THE RORSCHACH AND THE MINNESOTA MULTIPHASIC PERSONALITY INVENTORY: Results From Meta-Analyses

Results from meta-analyses have been widely cited to defend the validity of the Rorschach. However, the meta-analyses have been flawed. For example, one meta-analysis included results that were obtained by calculating correlations but not results that were obtained by conducting t tests or analyses of variance. When we reanalyzed the data from the most widely cited meta-analysis (Parker, Hanson, & Hunsley, 1988), we found that for confirmatory studies (also called convergent-validity studies), the Minnesota Multiphasic Personality Inventory (MMPI) explained 23% to 30% of the variance, whereas the Rorschach explained only 8% to 13% of the variance. These results indicate that the Rorschach is not as valid as the MMPI. The use of the Rorschach Inkblot Method continues to be controversial (e.g., Exner, 1996; Wood, Nezworski, & Stejskal, 1996a, 1996b). However, results from several meta-analyses seem to support the Rorschach (Atkinson, 1986; Atkinson, Quarrington, Alp, & Cyr, 1986; Parker, 1983; Parker, Hanson, & Hunsley, 1988). For example, based on the results for one of the meta-analyses, Parker et al. (1988) concluded that the convergent-validity estimates for the Rorschach and MMPI [Minnesota Multiphasic Personality Inventory] were not significantly different. . . . It appears that both the MMPI and Rorschach can be considered to have adequate psychometric properties if used for the purposes for which they were designed and validated. (abstract, p. 367) The results from the meta-analyses, especially the results from the meta-analysis conducted by Parker et al. (1988), have been widely cited to defend the validity of the Rorschach (e.g., Beutler & Davison, 1995; Ganellen, 1996; Masling, 1997; Shontz & Green, 1992; Weiner, 1996, 1997). It does seem unfair to argue that the Rorschach should not be used in clinical and forensic practice if meta-analyses have supported its use, especially in view of the fact that the meta-analysis conducted by Parker et al. (1988) was published in a highly prestigious journal (Psychological Bulletin) and has been reprinted as an illustrative example of meta-analysis in a book on methodology (Kazdin, 1992). Unfortunately, the meta-analyses that have been cited to support the Rorschach have been flawed. For example, in one meta-analysis, results on validity were not analyzed separately from results on reliability (Parker, 1983); in another, meta-analysis estimates were not made of the magnitude of effects (Atkinson et al., 1986); and in a third, meta-analysis effect sizes were estimated but statistical procedures were not used to aggregate estimates across studies (Atkinson, 1986). We describe problems with a fourth meta-analysis (Parker et al., 1988) in greater detail. For their meta-analysis, Parker et al. (1988) included Rorschach and MMPI studies that were published between 1970 and 1981 in the Journal of Personality Assessment and the Journal of Clinical Psychology. They placed results into an “unknown-validity” category if they “lacked a theoretical or empirical rationale” (p. 370) and into a “convergent-validity” category if the authors of an article described a theoretical reason for expecting a statistically significant result or if the intent of a study was to replicate a finding that was statistically significant in a previous study. Results for the unknown-validity category, reported as correlation coefficient equivalents of the average z scores, were .24 for the MMPI and .07 for the Rorschach. Results for the convergent-validity category were .46 for the MMPI and .41 for the Rorschach. A problem with the meta-analysis conducted by Parker et al. (1988) is that results that were obtained by calculating correlations were not pooled with results that were obtained by conducting t tests or F tests. Apparently, Parker et al. did not pool the correlation results with results obtained by conducting other statistical analyses because they believe that correlational analyses are more powerful. Indicative of this belief is the following statement made by Parker et al. (1988, p. 372): “Statistics should be rank ordered from the most to the least powerful as follows: correlations, ANOVAs [analyses of variance], and t tests.” However, textbooks on meta-analysis do not recommend that meta-analyses be based on correlation results alone. Instead, they recommend that F ratios, t tests, and chi-square values be included in analyses (e.g., Cooper & Hedges, 1994). If an independent variable is categorical (e.g., if the levels of the independent variable are normal subject and schizophrenic patient) and a dependent variable is continuous (e.g., a continuous Rorschach score), then the use of a t test or ANOVA is appropriate. Parker et al. (1988) did present results on convergent validity for studies that used correlational analysis, ANOVA, or t tests, but instead of pooling the data, they presented the results separately for the three different groups. However, unless the data are pooled, one cannot satisfactorally test for statistical significance or make estimates of level of validity. Other problems with the meta-analyses conducted by Parker et al. (1988) can also be described. First, assessment instruments sometimes shared methodological variance with the construct measures. For example, in one study (Last & Weiss, 1976), a Rorschach measure of ego strength was evaluated by comparing it with another Rorschach measure of ego strength. In this study, a large effect size may have been obtained because the two measures share method variance, not because either measure is a good measure of ego strength. Second, and even more damaging, in many of the studies coded by Parker et al., a small 402 Copyright © 1998 American Psychological Society VOL. 9, NO. 5, SEPTEMBER 1998 Address correspondence to Howard N. Garb, Behavioral Health (116A-H), VA Medical Center, 7180 Highland Dr., Pittsburgh, PA 15206-1297; e-mail: [email protected]. PSYCHOLOGICAL SCIENCE Howard N. Garb, Colleen M. Florio, and William M. Grove effect size meant that a test was valid. For example, in one study (Hersen & Greaves, 1971), a small effect size indicated that Rorschach productivity is not related to verbal reinforcement. Similarly, in many studies (e.g., Griffin, Finch, Edwards, & Kendall, 1976; Newmark, Newmark, & Cook, 1975), an abbreviated form of the MMPI was highly correlated with the standard form of the MMPI, but the effect size was estimated to be zero (apparently because statistical tests between the short form and regular form were typically nonsignificant). REANALYSIS OF THE PARKER ET AL. (1988) DATA